Dynamic Workflows and Advanced Data Management for Problem Solving Environments

Workflow management in problem solving environments (PSEs) is an emerging topic that aims to combine both data-oriented and execution-oriented views of scientific experiments, and closely integrate the processes underlying the practice of computational science with the software artifacts constituted by the PSE. This thesis presents a workflow management solution called BREW (BetteR Experiments through Workflow management) that provides functionality along four dimensions: components and installation management, experiment execution management, data management, and (full fledged) workflow management. BREW builds upon EMDAG, a first generation experiment management system designed at Virginia Tech which provided rudimentary facilities for supporting (only) the first two functionalities. BREW provides a complete dynamic workflow management solution wherein the PSE user can compose arbitrary scientific experiments and specify intended dynamic behavior of these experiments to an extent not previously possible. Along with the design details of the BREW system, this thesis identifies important tradeoffs underlying workflow management for PSEs, and presents two case studies involving large-scale data assimilation in bioinformatics experiments.